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The Student-Centered Arts-Learning Environ- 
ments project (SCALE Project) focused on 
professional development for teachers that 
enabled them to integrate arts into other 
curricular areas through a model of cross- 
discipline collaboration. Teachers prepared to 
lead elementary school students through the 
planning and implementation of an arts-based 
cross-curricular project, and to attend a theme- 
related concert performed by the West Virginia 
Symphony Orchestra (WVSO). 


During late spring 2012, staff from the WVSO 
and West Virginia Department of Education 
(WVDE) Office of Instruction recruited 16 schools 
in high poverty areas to participate in the SCALE 
Project during the following fall. Each school sent 
a team (including a regular education teacher, 
arts teacher, and a school administrator) to two 
face-to-face training sessions (in June and 
October 2012) and a webinar (August 2012). 
During the subsequent weeks, teams 
implemented their projects and prepared their 
students to participate in a performance 
(November 2012) of Maestro Grant Cooper's Boyz 
in the Wood, a contemporary children’s concert 
based on the story of Little Red Riding Hood. 


The overall goals of the SCALE Project included 
increases in (a) peer collaboration among 
teachers for  interdisciplinary instructional 
planning and implementation of arts integration; 
(b) teacher knowledge of arts integration and 
lesson design; and (c) student engagement in 
learning and in music and arts. The WVDE Office 
of Research with support from the Office of Early 
Learning, Office of Instruction, and WVSO 


West Virginia Department of Education, Office of Research 


undertook a collaborative program evaluation to 
measure progress toward those goals. The 
evaluation addressed five core evaluation 
questions (EQs) as described in the results 
section below. 


Methods 


The primary sources of data were the following: 


e The SCALE Project Professional Development 
Evaluation Survey was deployed to SCALE 
Project team members onsite at the end of 
their final training session in October 2012. 
We received 28 responses from the 32 
participants, for an 87.5% response rate. 


e The SCALE Project Implementation Rubric 
and Checklist was used from October 2012 
through early April 2012 by WVDE and 
WVSO staff/consultants to record observa- 
tions for 15 of the 16 schools in the project, 
representing 93.8%. 


e The Project SCALE School Survey (Pre 
Survey) was deployed to the entire school 
faculty in SCALE schools early in the fall 
before the implementation of SCALE projects 
and again the following February after 
projects had ended. Only respondents of the 
Pre Survey participated in the Post Survey. 
For the Pre Survey we received 164 responses 
from 371 teachers contacted (44%). Of those 
164 respondents to the Pre Survey, we 
received 111 Post Survey responses (68%). 


Descriptive statistics and qualitative data analysis 
were used to summarize data related to EQ1, 


including measures of central tendency and 
dispersion (e.g., mean/standard deviation) as well 
as percentages of respondents indicating specific 
perceptions of the SCALE program. Likewise for 
EQ2, we used descriptive statistics to summarize 
implementation fidelity data, describing the 
number and proportion of schools that met 
adequate fidelity on the core indicators noted 
above. To address EQ3, EQ4, and EQ5, we 
conducted a series of paired t tests whereby we 
ascertained if changes from _ pre-_ to 
postintervention were statistically significant and 
in the predicted direction, and we used Hedge’s g 
test to determine if changes observed were 
substantively important. 


Results 


Evaluation Question 1. What was the quality of 
the training and technical assistance that was 
provided to SCALE Project schools? 


Professional development offered to participants 
in the SCALE Project was well attended and well 
received. The overall ratings were remarkably 
high, with participants strongly agreeing about 
the high quality of the training and trainers on 
multiple measures, and approaching strong 
agreement about the high quality of the materials 
provided. The comments from participants also 
strongly praised the program, materials, and 
trainers. A few indicated they did not feel 
prepared to implement the project in their 
schools—that is, they still had lots of questions— 
after the summer professional development, but 
when they had completed all three training 
events, they indicated they were looking forward 
to using what they had learned back in their 
schools and classrooms. Overall, based on 
participant perceptions, SCALE Project team 
leaders were well trained to facilitate the program 
in their schools. 


Evaluation Question 2. How well did the SCALE 
Project schools implement the central 
components of the program? 


Rubric and benchmark data provided by West 
Virginia Symphony Orchestra (WVSO) and West 
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Virginia Department of Education (WVDE) 
personnel indicated that in most schools, the 
SCALE Project was well implemented (80%). As a 
group, rubric data indicated that schools saw the 
greatest level of implementation relative to 
student engagement in the arts and the lowest 
level of implementation relative to improving 
lesson design. Other data showed the following: 
(a) there was agreement or strong agreement that 
student engagement was high on all measures; 
(b) in the large majority of schools, at least six of 
10 role groups were involved in the project, with 
Title I teachers, physical education teachers, 
parents, and community members the most 
commonly not involved; (c) the arts were well 
integrated into the SCALE school projects; and 
(d) the large majority of schools implemented 
eight of nine major SCALE Project components. 


Implementation was far from even, however. 
Four schools had implementation rates above 
90% (high-implementation schools), while four 
had implementation rates below 75%, ranging 
down to 51% (low-implementation schools). For 
those that had lower levels of implementation, 
several issues seemed to be the most common: (a) 
forming a team; (b) arranging for it to meet 
regularly; (c) limited arts integration professional 
development for staff; (d) limited involvement of 
content areas other than the arts; or (e) 
involvement of only some, not all, of their 
classrooms. 


The unevenness in implementation did not seem 
related to previous experience with the program 
(i.e., all four low-implementation schools had 
previous experience with the SCALE Project, as 
did three of the four high-implementation 
schools), nor with schools’ involvement in the 
School Improvement Grant (SIG) program (half 
of the low implementation and half of the high 
implementation schools were SIG _ schools). 
Although it is not possible to say with certainty, 
successful implementation seemed related to the 
school’s commitment to the project—which 
included establishing an active, multidisciplinary 
team, strong collaboration, and involvement of 
the whole school. 


Evaluation Question 3. To what extent did the 
SCALE Project result in increased student 
engagement in music and the arts and in other 
content areas? 


SCALE school teacher surveys conducted before 
and after implementation of the project contained 
three subscales measuring educators’ perceptions 
of student affective, behavioral, and cognitive 
engagement in the arts and other content areas. 
All three subscales showed small improvements, 
although only one, behavioral engagement, 
approached statistical significance. When looking 
at individual survey items that made up the 
subscales, two interesting findings emerged— 
teachers reported small positive effects for 
students staying on task and for student 
motivation. 


When comparing schools with prior experience in 
SCALE with those new to the project, there were 
small but substantively important positive 
changes in both behavioral and _ cognitive 
engagement for schools new to SCALE. Looking 
at individual items, we found two of the strongest 
effects we detected in the study: Teachers in new- 
implementation schools reported higher levels of 
students staying on task at Post Survey than at 
Pre Survey and higher levels of students believing 
they were learning in their classes. For prior- 
implementation schools we found there were 
slightly lower perceptions of the extent to which 
students were excited about their schoolwork at 
Post Survey. 


As noted in the discussion of EQ2, there were 
large differences among the schools in their levels 
of implementation, so we looked at what bearing, 
if any, implementation fidelity had on changes 
between Pre and Post Survey. With respect to 
overall subscale scores, we found that educators 
in high-fidelity schools reported both higher 
overall behavioral and cognitive engagement 
among students at Post Survey than at Pre 
Survey. When we looked a little deeper at 
individual student engagement items, educators 
in high-fidelity schools at Post Survey reported 
(a) students stayed on task more, (b) preferred 
more challenging assignments, and (c) followed 
instructions better. 


Evaluation Question 4. To what extent did the 
SCALE Project impact culture and climate in 
participating schools? 


Our analysis of pre- and posttest responses to 
school culture items on the school survey revealed 
no statistically significant or substantively 
important changes from pre- to post-test data 
collections on any of the four subscales. These 
findings held when we disaggregated the 
responses by new or prior implementation and by 
high- and low-implementation fidelity. This is not 
surprising for such a brief intervention such as 
the SCALE Project. For an intervention that 
lasted only a few months, it would be unusual to 
see significant changes. 


Evaluation Question 5. To what extent did the 
SCALE Project result in improved lesson design 
and/or instructional delivery among 
participating schools? 


We constructed three subscales in the Project 
SCALE School Survey (Pre and Post Survey 
versions) for lesson design and _ instructional 
delivery to measure (a) integration of arts content 
into instruction, (b) collaboration among faculty 
role groups and community members in lesson 
design, and (c) collaboration among school-based 
teams in lesson planning. We found no 
statistically significant or substantively important 
changes at the subscale level for the whole group, 
nor for schools disaggregated by experience with 
the SCALE Project (new versus prior), nor for 
schools disaggregated by level of fidelity of 
implementation (high versus low). We _ also 
disaggregated responses by content area (arts 
versus other content areas), and again did not 
find changes to report. When looking at 
individual items, however, there were several 
interesting—albeit small effects at Post Survey: 


e Teachers from new-implementation schools 
reported more collaboration with community 
members in the development of their lessons, 
and more integration of dance/movement 
strategies into their own instruction. 


e Non-arts elementary teachers reported more 
integration of dance/movement strategies 
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into their own instruction, and more 
collaboration with arts teachers. 


e Teachers from high-implementation-fidelity 
schools reported less integration of creative 
writing instructional strategies, and more 
integration of dance instructional strategies. 


Limitations of the Study 


Several limitations of this study bear mentioning. 
First, all data are self-reported and thus subject to 
various threats to validity, such as_ social 
desirability bias (when respondents provide 
overly positive responses to a survey or 
questionnaire due to their desire to be viewed 
favorably) or nonresponse’ bias (when 
respondents who elect not to participate in a 
survey differ in a meaningful way with those who 
do). We encountered one technical difficulty, 
when we unintentionally excluded an item in one 
of the subscales drawn from the POSC 
instrument. To compensate for the missing item, 
we applied a multiplier. There is a _ small 
possibility that the validity of the measure was 
affected, but there was little or no change in these 
measures, so this error did not impact our 
findings in any meaningful way. Another 
limitation of our study involves our inability to 
draw school-level conclusions regarding the 
impact of the various projects implemented by 
schools, because we did not receive a large 
number of completed surveys from many schools 
to make such disaggregation possible. 


Recommendations 


We make the following recommendations based 
upon our results: 


e To the extent possible, we recommend 
continuing this project. Educators appear to 
perceive positive benefits of the program for 
their students especially in the areas of 
cognitive and behavioral engagement. These 
are important outcomes that could lead to 
improved student achievement if sustained. 


e Efforts should be made to sustain initial 
excitement so that prior implementation 
schools can continue to realize benefits. In 
this study, we found that new implementation 
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schools realized more positive outcomes than 
prior-implementation schools. The 
excitement factor in new schools could have 
contributed to these findings. 


Encourage and support full implementation 
of all components of the program. We found 
that higher-than-average implementation 
fidelity schools experienced more positive 
outcomes than those schools that did not 
implement many components with fidelity. 
We found no significant changes for lower- 
than-average implementation schools. That is, 
while failure to implement the program as 
intended is not necessarily associated with 
negative outcomes, it does _ potentially 
maintain the status quo. Program staff should 
use these results as a catalyst for participating 
schools, to illustrate that a school’s level of 
commitment can make or break the project. 


Ensure that schools participating in the 
project build in sufficient common planning 
time to support the necessary collaboration. 
This time is essential to ensure the school’s 
project is implemented with fidelity and 
achieves the intended school-wide outcomes. 
Administrator support is critical in this regard 
and should be discussed early on in the 
project. 


Develop strategies to ensure that once the 
school project concludes, the faculty does not 
return to business as usual. One strategy may 
be finding ways to sustain the momentum of 
the project—that is to continue on with other 
collaborative projects that integrate various 
content areas, including the arts. Another 
strategy may involve addressing beliefs that 
time spent on a project such as SCALE is time 
taken away from improving test scores in 
mathematics and _ reading/language arts. 
Helping educators understand the strong 
connections between high student cognitive 
and behavior engagement—as seen in many of 
the SCALE schools this year—and high 
student achievement could help reduce the 
anxiety felt about making the sorts of changes 
in lesson planning and instruction that the 
SCALE project encouraged. 


